fix: stop CSP allow-lists triggering challenge detection (#264) - #266

Fixes #264.

The problem

webcmd web fetch wrongly decides a normal page is a bot-challenge page, and gives up on it.

The check that spots challenge pages searched every response header for words like cloudflare, recaptcha and captcha. Plenty of healthy sites list those domains in their content-security-policy header — that header is just an allow-list of third parties, not a statement about this response.

Hacker News is one of them. It returns a healthy 200 with the full page, gets labelled a challenge, retries through the whole impit ladder for nothing, and then fails:

$ webcmd web fetch --url https://news.ycombinator.com
CliError: The site blocked non-browser fetches.
  hint: Use webcmd web fetch-browser for this URL.

The page was never blocked. Any site whose CSP names a CDN or reCAPTCHA — a large share of the web — paid two wasted fetches and then a misleading FETCH_BLOCKED.

What I changed

src/fetch/classify.ts:

1. Only look at headers that describe this response: server, cf-mitigated, cf-chl-*, x-datadome*, set-cookie. content-security-policy, report-to and link are ignored — they are third-party allow-lists and prove nothing.
2. A 200 now needs evidence in the body before it counts as a challenge. server: cloudflare on a page that was actually served is not a challenge; a real interstitial still says "Just a moment" in the body, and that still matches.
3. Body matching is unchanged.

Proof

$ node dist/src/main.js web fetch --url https://news.ycombinator.com
# Hacker News   ← front page, plain tier, no ladder

• npx vitest run src/fetch/ → 22 passed. npm run typecheck clean.
• New tests cover: the CSP false positive, a header-only 200, and that a real 403 + server: cloudflare is still classified as a challenge (no under-detection).

🤖 Generated with Claude Code